Neural Networks

Popularly referred to as the GPS of the brain, the hippocampus has a variety of neurons that encode spatial properties of the environment. These spatial cells of the hippocampus may be broadly placed under two categories - those that encode spatial locations (e.g. place cells, grid cells etc) and those that encode spatial objects (eg. Object-sensitive cells. Object-trace cells etc). There are computational models that explain emergence of specific types of spatial cells, but it is challenging to construct integrative models that can demonstrate the emergence of the complete range of spatial cells both space and object type. We present a simple, unified computational model that explains the emergence of a wide variety of object- and spatially-sensitive neurons in the hippocampus. The model is essentially a deep neural network that combines visual and path integration information. The visual information is received by a part of the model that is analogous to Lateral Entorhinal Cortex (LEC) and path integration information is received by a layer analogous to Medial Entorhinal Cortex (MEC). In order to arrive at a consistent estimate of position, LEC and MEC in the model are connected laterally using a Graph Neural Network. The model is trained to predict position, orientation and reward of a simulated agent. The agent explores a box-like environment with colored walls and objects on the floor and is rewarded based on its encounters with objects. The model demonstrates the emergence of the following 7 types of spatial and object cells - place, grid, border, object, object-sensitive, object-vector and, object-trace cells. The model findings compare favorably with a large body of experimental literature on hippocampal spatial cells.

Storing complex correlated memories is significantly more efficient when memories are recoded to obtain compressed representations. Previous work has shown that compression can be implemented in a simple neural circuit, which can be described as a sparse autoencoder. The activity of the encoding units in these models recapitulates the activity of hippocampal neurons recorded in multiple experiments. However, these investigations have assumed that the level of sparsity is fixed and that inputs have the same statistics and, hence, that they are uniformly compressible. In contrast, biological agents encounter environments with vastly different memory demands and compressibility. Here, we investigate whether the compressibility of inputs determines optimal sparsity in sparse autoencoders. We find 1) that as the compressibility of inputs increases, the optimal coding level decreases, 2) that the desired coding level diverges from the observed coding level as a function of both memory demand and input compressibility, and 3) that optimal memory capacity is achieved when sparsity is weakly enforced. In addition, we characterize how sparsity and the strength of sparsity enforcement jointly control optimal performance. These results provide predictions for how sparsity in the hippocampus should change in response to environmental statistics and theoretical grounds for why sparsity is dynamically tuned in the brain.

Novelty detection, also known as familiarity discrimination or recognition memory, refers to the ability to distinguish whether a stimulus has been seen before. It has been hypothesized that novelty detection can naturally arise within networks that store memory or learn efficient neural representation, because these networks already store information on familiar stimuli. However, computational models instantiating this hypothesis have not been shown to reproduce high capacity of human recognition memory, so it is unclear if this hypothesis is feasible. This paper demonstrates that predictive coding, which is an established model previously shown to effectively support representation learning and memory, can also naturally discriminate novelty with high capacity. Predictive coding model includes neurons encoding prediction errors, and we show that these neurons produce higher activity for novel stimuli, so that the novelty can be decoded from their activity. Moreover, the hierarchical predictive coding networks uniquely perform novelty detection at varying abstraction levels across the hierarchy, i.e., they can detect both novel low-level features, and novel higher-level objects. Overall, we unify novelty detection, associative memory, and representation learning within a single computational framework.

Same-different visual reasoning is a basic skill central to abstract combinatorial thought. This fact has lead neural networks researchers to test same-different classification on deep convolutional neural networks (DCNNs), which has resulted in a controversy regarding whether this skill is within the capacity of these models. However, most tests of same-different classification rely on testing on images that come from the same pixel-level distribution as the testing images, yielding the results inconclusive. In this study we tested relational same-different reasoning DCNNs. In a series of simulations we show that DCNNs are capable of visual same-different classification, but only when the test images are similar to the training images at the pixel-level. In contrast, even when there are only subtle differences between the testing and training images, the performance of DCNNs could drop to chance levels. This is true even when DCNNs training regime included a wide distribution of images or when they were trained in a multi-task setup in which training included an additional relational task with test images from the same pixel-level distribution.

MotivationReconstructing neuron-level brain circuit network is a universally recognized formidable task. A significant impediment involves discerning the intricate interconnections among multitudinous neurons in a complex brain network. However, the majority of current methodologies only rely on learning local visual synapse features while neglecting the incorporation of comprehensive global topological connectivity information. In this paper, we consider the perspective of network connectivity and introduce graph neural networks to learn the topological features of brain networks. As a result, we propose Neuronal Circuit Prediction Network (NCPNet), a simple and effective model to jointly learn node structural representation and neighborhood representation, constructing neuronal connection pair feature for inferring neuron-level connections in a brain circuit network. ResultsWe use a small number of connections randomly selected from a single brain circuit network as training data, expecting NCPNet to extrapolate known connections to unseen instances. We evaluated our model on Drosophila connectome and C. elegans worm connectome. The numerical results demonstrate that our model achieves a prediction accuracy of 91.88% for neuronal connections in the Drosophila connectome when utilizing only 5% of known connections. Similarly, under the condition of 5% known connections in C. elegans, our model achieves an accuracy of 93.79%. Additional qualitative analysis conducted on the learned representation vectors of Kenyon cells indicates that NCPNet successfully acquires meaningful features that enable the discrimination of neuronal sub-types. Our project is available at https://github.com/mxz12119/NCPNet.

Deep convolutional neural networks (CNNs) are powerful computational tools for a large variety of tasks (Goodfellow, 2016). Their architecture, composed of layers of repeated identical neural units, draws inspiration from visual neuroscience. However, biological circuits contain a myriad of additional details and complexity not translated to CNNs, including diverse neural cell types (Tasic, 2018). Many possible roles for neural cell types have been proposed, including: learning, stabilizing excitation and inhibition, and diverse normalization (Marblestone, 2016; Gouwens, 2019). Here we investigate whether neural cell types, instantiated as diverse activation functions in CNNs, can assist in the feed-forward computational abilities of neural circuits. Our heterogeneous cell type networks mix multiple activation functions within each activation layer. We assess the value of mixed activation functions by comparing image classification performance to that of homogeneous control networks with only one activation function per network. We observe that mixing activation functions can improve the image classification abilities of CNNs. Importantly, we find larger improvements when the activation functions are more diverse, and in more constrained networks. Our results suggest a feed-forward computational role for diverse cell types in biological circuits. Additionally, our results open new avenues for the development of more powerful CNNs.

A key driver of mammalian intelligence is the ability to represent incoming sensory information across multiple abstraction levels. For example, in the visual ventral stream, incoming signals are first represented as low-level edge filters and then transformed into high-level object representations. These same hierarchical structures routinely emerge in artificial neural networks (ANNs) trained for image/object recognition tasks, suggesting that a similar process might underlie biological neural networks. However, the classical ANN training algorithm, backpropagation, is considered biologically implausible, and thus several alternative biologically plausible methods have been developed. For instance, several cortical-inspired ANNs in which the apical dendrite of a pyramidal neuron encodes top-down prediction signals have been proposed. In this case, akin to theories of predictive coding, a prediction error can be calculated locally inside each neuron for updating its incoming weights. Notwithstanding, from a neuroscience perspective, it is unclear whether neurons could compare their apical vs. somatic spiking activities to compute prediction errors. Here, we propose a solution to this problem by adapting the framework of the apical-somatic prediction error to the temporal domain. In particular, we show that if the apical feedback signal changes the postsynaptic firing rate, we can use differential Hebbian updates, a rate-based version of the classical spiking time-dependent plasticity (STDP) updates. To the best of our knowledge, this is the first time a cortical-like deep ANN has been trained using such time-based learning rules. Overall, our work removes a key requirement of biologically plausible models for deep learning that does not align with plasticity rules observed in biology and proposes a learning mechanism that would explain how the timing of neuronal activity can allow supervised hierarchical learning.

Visual object and scene recognition have been extensively studied, but separately. We here propose that the two processes could be intrinsically linked in the neural system. We developed a Joint Residual Variational Autoencoder (JRVAE) with two networks: VAE1 for coarse scene recognition and VAE2 for object recognition using residuals from VAE1s reconstructions. Our model demonstrates emergent functional specialization when conditioned on information reduction in peripheral vision, with quantitative analysis confirming VAE1 excels at the representation of scenes while VAE2 specializes in that of objects. This architecture naturally implements figure-ground segmentation and aligns with neurobiological evidence of distinct cortical pathways. Our findings suggest residual computation enables joint visual processing that mirrors human perceptions coarse-to-fine principle in perception.

In the search for biologically plausible but mathematically precise theories of learning in the brain, recent studies have begun to investigate how key assumptions underlying algorithms for supervised learning in artificial neural networks can be relaxed in biologically plausible ways. Turning to unsupervised learning, we develop biologically more plausible variants of the restricted Boltzmann machine (RBM), and benchmark their performance on MNIST. We show that RBMs with asymmetric connectivity can still be successfully trained with contrastive divergence, even if no two units are reciprocally connected. Furthermore, RBMs are able to learn if the forward, visible-to-hidden layer weights are kept constant and only the backward, hidden-to-visible layer weights are updated. These findings indicate that neural networks with biologically plausible connectivity support contrastive learning.

Biological neuronal networks (BNNs) are a source of inspiration and analogy making for researchers that focus on artificial neuronal networks (ANNs). Moreover, neuroscientists increasingly use ANNs as a model for the brain. Despite certain similarities between these two types of networks, important differences can be discerned. First, biological neural networks are sculpted by evolution and the constraints that it entails, whereas artificial neural networks are engineered to solve particular tasks. Second, the network topology of these systems, apart from some analogies that can be drawn, exhibits pronounced differences. Here, we examine strategies to construct recurrent neural networks (RNNs) that instantiate the network topology of brains of different species. We refer to such RNNs as bio-instantiated. We investigate the performance of bio-instantiated RNNs in terms of: i) the prediction performance itself, that is, the capacity of the network to minimize the desired function at hand in test data, and ii) speed of training, that is, how fast during training the network reaches its optimal performance. We examine bio-instantiated RNNs in working memory tasks where task-relevant information must be tracked as a sequence of events unfolds in time. We highlight the strategies that can be used to construct RNNs with the network topology found in BNNs, without sacrificing performance. Despite that we observe no enhancement of performance when compared to randomly wired RNNs, our approach demonstrates how empirical neural network data can be used for constructing RNNs, thus, facilitating further experimentation with biologically realistic network topologies, in contexts where such aspect is desired.

How the brain modifies synapses to improve the performance of complicated networks remains one of the biggest mysteries in neuroscience. Canonical models suppose synaptic weights change according to pre- and post-synaptic activities (i.e., local plasticity rules), implementing gradient-descent algorithms. However, the lack of experimental evidence to confirm these models suggests that there may be important ingredients neglected by these models. For example, heterosynaptic plasticity, non-local rules mediated by inter-cellular signaling pathways, and the biological implementation of evolutionary algorithms (EA), another machine-learning paradigm that successfully trains large-scale neural networks, are seldom explored. Here we propose and systematically investigate an EA model of brain learning with non-local rules alone. Specifically, a population of agents are represented by different information routes in the brain, whose task performances are evaluated through gating on individual routes alternatively. The selection and reproduction of agents are realized by dopamine-guided heterosynaptic plasticity. Our EA model provides a framework to re-interpret the biological functions of dopamine, meta-plasticity of dendritic spines, memory replay, and the cooperative plasticity between the synapses within a dendritic neighborhood from a new and coherent aspect. Neural networks trained with the model exhibit analogous dynamics to the brain in cognitive tasks. Our EA model manifests broad competence to train spiking or analog neural networks with recurrent or feedforward architecture. Our EA model also demonstrates its powerful capability to train deep networks with biologically plausible binary weights in MNIST classification and Atari-game playing tasks with performance comparable with continuous-weight networks trained by gradient-based methods. Overall, our work leads to a fresh understanding of the brain learning mechanism unexplored by local rules and gradient-based algorithms.

Gap junctions in the retina are electrical synapses, which strength is regulated byambient light conditions. Such tunable synapses are crucial for the denoising function of the early visual system. However, it is unclear that how the plastic gap junction network processes unknown noise, specifically how this process works synergistically with the brains higher visual centers. Inspired by the electrically coupled photoreceptors, we develop a computational model of the gap junction filter (G-filter). We show that G-filter is an effective blind denoiser that converts different noise distributions into a similar form. Next, since deep convolutional neural networks (DCNNs) functionally reflect some intrinsic features of the visual cortex, we combine G-filter with DCNNs as retina and ventral visual pathways to investigate the relationship between retinal denoising processing and the brains high-level functions. In the image denoising and reconstruction task, G-filter dramatically improve the classic deep denoising convolutional neural network (DnCNN)s ability to process blind noise. Further, we find that the gap junction strength of the G-filter modulates the receptive field of DnCNNs output neurons by the Integrated Gradients method. At last, in the image classification task, G-filter strengthens the defense of state-of-the-arts DCNNs (ResNet50, VGG19 and InceptionV3) against blind noise attacks, far exceeding human performance when noise is large. Our results indicate G-filter significantly enhance DCNNs ability on various blind denoising tasks, implying an essential role for retina gap junction networks in high-level visual processing.

While primates are primarily visual animals, how visual information is processed on its way to memory structures and contributes to the generation of visuospatial behaviors is poorly understood. Recent imaging data demonstrate the existence of scene-sensitive areas in the dorsal visual path that are likely to combine visual information from successive egocentric views, while behavioral evidence indicates the memory of surrounding visual space in extraretinal coordinates. The present work focuses on the computational nature of a panoramic representation that is proposed to link visual and mnemonic functions during natural behavior. In a spiking neural network model of the dorsal visual path it is shown how time-integration of spatial views can give rise to such a representation and how it can subsequently be used to perform memory-based spatial reorientation and visual search. More generally, the model predicts a common role of view-based allocentric memory storage in spatial and non-spatial mnemonic behaviors.

AO_SCPLOWBSTRACTC_SCPLOWThe units in artificial neural networks (ANNs) can be thought of as abstractions of biological neurons, and ANNs are increasingly used in neuroscience research. However, there are many important differences between ANN units and real neurons. One of the most notable is the absence of Dales principle, which ensures that biological neurons are either exclusively excitatory or inhibitory. Dales principle is typically left out of ANNs because its inclusion impairs learning. This is problematic, because one of the great advantages of ANNs for neuroscience research is their ability to learn complicated, realistic tasks. Here, by taking inspiration from feedforward inhibitory interneurons in the brain we show that we can develop ANNs with separate populations of excitatory and inhibitory units that learn just as well as standard ANNs. We call these networks Dales ANNs (DANNs). We present two insights that enable DANNs to learn well: (1) DANNs are related to normalization schemes, and can be initialized such that the inhibition centres and standardizes the excitatory activity, (2) updates to inhibitory neuron parameters should be scaled using corrections based on the Fisher Information matrix. These results demonstrate how ANNs that respect Dales principle can be built without sacrificing learning performance, which is important for future work using ANNs as models of the brain. The results may also have interesting implications for how inhibitory plasticity in the real brain operates.

Communication subspaces have recently been identified as a promising mechanism for selectively routing information between brain areas. In this study, we explored whether communication sub-spaces develop with training in artificial neural networks (ANNs) and explored differences across connection types. Specifically, we analyzed the subspace angles between activations and weights in ResNet-50 before and after training. We found that activations were more aligned to the weight layers after training, although this effect decreased in deeper layers. We also analyzed the angles between pairs of weight layers. We found that for all branching, direct, and skip connections, weight layer pairs were more geometrically aligned in trained versus untrained models throughout the entire network. These findings indicate that such alignment is essential for the proper functioning of deep networks and highlights the potential to enhance training efficiency through pre-alignment. In biological data, our results motivate further exploration into whether learning induces similar subspace alignment.

Biological neurons integrate their inputs on dendrites using a diverse range of non-linear functions. However the majority of artificial neural networks (ANNs) ignore biological neurons structural complexity and instead use simplified point neurons. Can dendritic properties add value to ANNs? In this paper we investigate this question in the context of continual learning, an area where ANNs suffer from catastrophic forgetting (i.e., ANNs are unable to learn new information without erasing what they previously learned). We propose that dendritic properties can help neurons learn context-specific patterns and invoke highly sparse context-specific subnetworks. Within a continual learning scenario, these task-specific subnetworks interfere minimally with each other and, as a result, the network remembers previous tasks significantly better than standard ANNs. We then show that by combining dendritic networks with Synaptic Intelligence (a biologically motivated method for complex weights) we can achieve significant resilience to catastrophic forgetting, more than either technique can achieve on its own. Our neuron model is directly inspired by the biophysics of sustained depolarization following dendritic NMDA spikes. Our research sheds light on how biological properties of neurons can be used to solve scenarios that are typically impossible for traditional ANNs to solve.

The brains distributed architecture has inspired numerous artificial intelligence (AI) systems, particularly through its neocortical organization. However, current AI approaches largely overlook a crucial aspect of biological intelligence: active sensing - the deliberate movement of sensory organs to explore the environment. To explore how sensor movement impacts behavior in image classification tasks, we introduce the Active Neural Cellular Automata (ANCA), a neocortex-inspired model with movable sensors. Active sensing naturally emerges in the ANCA, with belief-informed exploration and attentive behavior to salient information, without adding explicit attention mechanisms. Active sensing both simplifies classification tasks and leads to a highly scalable system. This enables ANCAs to be smaller than the image size without losing information and enables fault tolerance to damaged sensors. Overall, our work provides insight to how distributed architectures can interact with movement, opening new avenues for adaptive AI systems in embodied agents.

Perceptual illusions are widely used to study brain processing, and are essential for elucidating underlying function. Successful brain models should then also be able to reproduce these illusions. Some of the most successful models for vision are several variants of Deep Neural Networks (DNNs). These models can classify images with human-level accuracy, and many behavioral and activation measurements correlate well with humans and animals. For several networks it was also shown that they can reproduce some human illusions. However, this was typically done for a limited number of networks. In addition, it remains unclear whether the presence of illusions is linked to either how accurate or brain-like the DNNs are. Here, we consider the scintillating grid illusion, to which two DNNs have been shown to respond as if they are impacted by the illusion. We develop a measure for measuring Illusion Strength based on model activation correlations, which takes into account the difference in Illusion Strength between illusion and control images. We then compare the Illusion Strength to both model performance (top-1 ImageNet), and how well the model explains brain activity (Brain-score). We show that the illusion was measurable in a wide variety of networks (41 out of 51). However, we do not find a strong correlation between Illusion Strength and Brain-Score, nor performance. Some models have strong illusion scores but not Brain-Score, or vice-versa, but no model does both well. Finally, this differs strongly between model types, particularly between convolutional and transformer-based architectures, with transformers having low illusion scores. Overall, our work shows that Illusion Strength measures an important metric to consider for assessing brain models, and that some models could still be missing out on some processing important for brain functioning.

The neocortex can be viewed as a tapestry consisting of variations of rather stereotypical local cortical microcircuits. Hence understanding how these microcircuits compute holds the key to understanding brain function. Intense research efforts over several decades have culminated in a detailed model of a generic cortical microcircuit in the primary visual cortex from the Allen Institute. We are presenting here methods and first results for understanding computational properties of this largescale data-based model. We show that it can solve a standard image-change-detection task almost as well as the living brain. Furthermore, we unravel the computational strategy of the model and elucidate the computational role of diverse subtypes of neurons. Altogether this work demonstrates the feasibility and scientific potential of a methodology based on close interaction of detailed data and large-scale computer modelling for understanding brain function.

Object recognition requires flexible and robust information processing, especially in view of the challenges posed by naturalistic visual settings. The ventral stream in visual cortex is provided with this robustness by its recurrent connectivity. Recurrent deep neural networks (DNNs) have recently emerged as promising models of the ventral stream, surpassing feedforward DNNs in the ability to account for brain representations. In this study, we asked whether recurrent DNNs could also better account for human behaviour during visual recognition. We assembled a stimulus set that included manipulations that are often associated with recurrent processing in the literature, like occlusion, partial viewing, clutter, and spatial phase scrambling. We obtained a benchmark dataset from human participants performing a categorisation task on this stimulus set. By applying a wide range of model architectures to the same task, we uncovered a nuanced relationship between recurrence, model size, and performance. While recurrent models reach higher performance than their feedforward counterpart, we could not dissociate this improvement from that obtained by increasing model size. We found consistency between humans and models patterns of difficulty across the visual manipulations, but this was not modulated in an obvious way by the specific type of recurrence or size added to the model. Finally, depth/size rather than recurrence makes model confusion patterns more human-like. Contrary to previous assumptions, our findings challenge the notion that recurrent models are better models of human recognition behaviour than feedforward models, and emphasise the complexity of incorporating recurrence into computational models.